Bad Memory Leak Bug with triggers - DOORS 8 and 9 affected

Hi all,

I stumbled upon a bad memory leak with triggers, that will probably affect scripts, that evaluate a lot of modules containing 'open' triggers. The problem is, that the stack memory, that a trigger script reserves will stay in memory, even after the module is closed. The following script demonstrates the problem:
  

string sTriggerName = "TestTrigger" 
 
string sModuleName = "Trigger Problem Module"
 
// Delete the module if it already exists
Item I = item sModuleName
if (!null I) {
    if (!confirm "About to delete " sModuleName ". OK?") { ack "Abort."; halt }
        if (!null module I) close (module I)
        delete I
        purge I
}
 
 
// This problem affects the trigger code itself 
Module mod = create (sModuleName, "", "", 1, true)
ModName_ mn = module mod
current = mod
 
// Install a post open trigger on the module
Trigger T = trigger (sTriggerName, module, post, open, 10, "\npragma stack, 10000000\nprint \"Inside Trigger...\"\n")
if (null T) error "Trigger " sTriggerName " could not be installed!\n"
 
save mod; close (mod, false)
 
ack "After creating the module - Look at the memory of DOORS.exe. About to run the trigger."
 
mod = read(fullName mn, false, true) 
 
ack "Trigger was executed. Look at the memory of DOORS.exe.\n\n" // -
    "The trigger stack is kept alive! 40MB of memory are gone.\n" // -
    "About to execute the DXL Attribute!"
 
close (mod)
 
ack "The module is closed now, but the stack memory of the trigger is leaked!\n\n" // -
    "The memory of the trigger DXL should have been freed after closing the module!"

 


That means that if you have triggers installed, that include a large library, then the stack space for the complete library will be leaked, every time you open a module containing the trigger. For us we had an trigger, that did some status checks on the module after opening the module which was installed on project level. The trigger included a lot of libraries containing business logic. This lead to the effect, that everytime we opened a module in the project 1MB of memory was leaked. This made our traceability scripts die, when trying to make a project traceability.

The resolution was to wrap the trigger code in eval_ as described in the tutorial, so the library was included in the eval and then get the trigger result back from the eval context.

Regards, Mathias

 


Mathias Mamsch - Thu Mar 28 04:22:44 EDT 2013

Re: Bad Memory Leak Bug with triggers - DOORS 8 and 9 affected
llandale - Thu Mar 28 14:28:14 EDT 2013

Mmmmmmm. Ran your code and confirmed your observation. However, when I changed the trigger to a post-attr-save trigger there was no such wasted space when the module was opened; but when I manualled edited it and changed the Prefix, your 40k waste happened. So it is running the trigger, not its existance.

This code run from the DXL window causes 80mb of wasted space during the ack; but reclaimed afterwards:
  • pragma stack, 10000000 // 10,000,000
  • ack "hello"

I notice 80mb is double the wasted space of the trigger 40mb.

I don't notice unclaimed space with a normal trigger, but that's because the default stack size is 4096 which should waste only "16kb" which is hardly noticeable in task manager.

Not following your references to "library". I have a massive library and don't see ..err.. notice these leaks. Seems to me the leak is associated with the "pragma stack" commands, not the size of #included files.

Does not "stack" represent "depth of function calls" (and specifically not the number of such calls) and really only matters when you have recursion? And if so, since triggers are usually pretty basic, just put "pragma stack, 200" in the triggers?

Does this wasted space mean triggers are run in the top context, or something like that?

-Louie

Re: Bad Memory Leak Bug with triggers - DOORS 8 and 9 affected
Mathias Mamsch - Fri Mar 29 04:39:13 EDT 2013

llandale - Thu Mar 28 14:28:14 EDT 2013
Mmmmmmm. Ran your code and confirmed your observation. However, when I changed the trigger to a post-attr-save trigger there was no such wasted space when the module was opened; but when I manualled edited it and changed the Prefix, your 40k waste happened. So it is running the trigger, not its existance.

This code run from the DXL window causes 80mb of wasted space during the ack; but reclaimed afterwards:

  • pragma stack, 10000000 // 10,000,000
  • ack "hello"

I notice 80mb is double the wasted space of the trigger 40mb.

I don't notice unclaimed space with a normal trigger, but that's because the default stack size is 4096 which should waste only "16kb" which is hardly noticeable in task manager.

Not following your references to "library". I have a massive library and don't see ..err.. notice these leaks. Seems to me the leak is associated with the "pragma stack" commands, not the size of #included files.

Does not "stack" represent "depth of function calls" (and specifically not the number of such calls) and really only matters when you have recursion? And if so, since triggers are usually pretty basic, just put "pragma stack, 200" in the triggers?

Does this wasted space mean triggers are run in the top context, or something like that?

-Louie

The space is wasted every time the trigger is 'interpreted', just before it is run. Short explanation of stack space:

If you have a DXL that declares two global integer variables:
  

int a 
int b
print "Hallo"

 


Then when the DXL parser translates the code to the DXL VM instructions, it will create a DXL context for the code, make a stack memory, and in there reserve space for the global variable a and b (8 bytes). Then it will reserve an additional stack space for dynamic variables and function parameters, variables and such, that are created at runtime. The DXL context itself however stores much more information, like for example the translated code, symbol tables, etc.

That means if in your trigger code you do:

 

 

  

#include <big_library_1.inc>
#include <big_library_2.inc>
// ...
#include <big_library_n.inc>
 
int var = call_some_function() 
if (var != 0) set trigPreConFail



Then the DXL context of the trigger can become pretty big, because all global variables of the big libraries are reserved on the stack, symbol tables for all functions, etc. will be created, the generated code, etc.

The problem now is, that all information of the complete DXL context seems not to be cleaned up properly after the trigger is run. Worse the trigger is not reused, but every time it is executed, this will happen again.

Of course one could think of stack optimization, especially look critically at "pragma stack" instructions inside libraries that might be included in such triggers. I cannot imagine that global variables even of the most complex library will result in large waste of space. Still we had 1MB for one of our triggers. So I am not completely sure if my theory explains this completely. We also notices a strange memory effect with those triggers and the execution of DXL attributes, but I could not reproduce it in the example code.

Regards, Mathias



 

 

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

Re: Bad Memory Leak Bug with triggers - DOORS 8 and 9 affected
llandale - Tue Apr 02 10:05:50 EDT 2013

Mathias Mamsch - Fri Mar 29 04:39:13 EDT 2013

The space is wasted every time the trigger is 'interpreted', just before it is run. Short explanation of stack space:

If you have a DXL that declares two global integer variables:
  

int a 
int b
print "Hallo"

 


Then when the DXL parser translates the code to the DXL VM instructions, it will create a DXL context for the code, make a stack memory, and in there reserve space for the global variable a and b (8 bytes). Then it will reserve an additional stack space for dynamic variables and function parameters, variables and such, that are created at runtime. The DXL context itself however stores much more information, like for example the translated code, symbol tables, etc.

That means if in your trigger code you do:

 

 

  

#include <big_library_1.inc>
#include <big_library_2.inc>
// ...
#include <big_library_n.inc>
 
int var = call_some_function() 
if (var != 0) set trigPreConFail



Then the DXL context of the trigger can become pretty big, because all global variables of the big libraries are reserved on the stack, symbol tables for all functions, etc. will be created, the generated code, etc.

The problem now is, that all information of the complete DXL context seems not to be cleaned up properly after the trigger is run. Worse the trigger is not reused, but every time it is executed, this will happen again.

Of course one could think of stack optimization, especially look critically at "pragma stack" instructions inside libraries that might be included in such triggers. I cannot imagine that global variables even of the most complex library will result in large waste of space. Still we had 1MB for one of our triggers. So I am not completely sure if my theory explains this completely. We also notices a strange memory effect with those triggers and the execution of DXL attributes, but I could not reproduce it in the example code.

Regards, Mathias



 

 

 

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

 

I've have ..
  1. some report scripts that open 1000s of modules
  2. had a small database wide post-module-open trigger running for years
  3. newer modules tend to have lots of attr-DXL
... and haven't noticed a problem; I guess I'll start noticing now. Or maybe I cleverly turn off running triggers and forgot...

So, do I understand your suggestion correctly here for Triggers:?
Instead of having this in a trigger:
  • #include <MyTriggers\ThisSpecificTrigger.dxl>
have this:
  • eval_("#include <MyTriggers\\ThisSpecificTrigger.dxl>")

And if so, then instead of this Attr-DXL code:
  • #include <MyAttrDXL\ThisSpecificAttr-DXL.dxl>
Have this:
  • int oRef = addr_ obj
  • eval_("Object oDeRef = addr_ " oRef "obj = oDeRef;#include <MyAttrDXL\\ThisSpecificAttr-DXL.dxl>")
(..err.. after you correct the syntax anyway..)

-Louie

I have noticed that Attr-DXL is interpreted
  1. when you open the module, even if you don't access an attr-dxl value
  2. when you modify the code and then access a value

That means if you attr-DXL is simply an #include statement, when debugging you have to modify the code and then modify and save the attr-DXL. Simply re-saving is inadequate so I tend to add or remove a space to the end of the #include statement. Thus att-DXL doesn't waste space based on how often it runs.

Re: Bad Memory Leak Bug with triggers - DOORS 8 and 9 affected
SystemAdmin - Wed Apr 03 03:41:55 EDT 2013

llandale - Tue Apr 02 10:05:50 EDT 2013
I've have ..

  1. some report scripts that open 1000s of modules
  2. had a small database wide post-module-open trigger running for years
  3. newer modules tend to have lots of attr-DXL
... and haven't noticed a problem; I guess I'll start noticing now. Or maybe I cleverly turn off running triggers and forgot...

So, do I understand your suggestion correctly here for Triggers:?
Instead of having this in a trigger:
  • #include <MyTriggers\ThisSpecificTrigger.dxl>
have this:
  • eval_("#include <MyTriggers\\ThisSpecificTrigger.dxl>")

And if so, then instead of this Attr-DXL code:
  • #include <MyAttrDXL\ThisSpecificAttr-DXL.dxl>
Have this:
  • int oRef = addr_ obj
  • eval_("Object oDeRef = addr_ " oRef "obj = oDeRef;#include <MyAttrDXL\\ThisSpecificAttr-DXL.dxl>")
(..err.. after you correct the syntax anyway..)

-Louie

I have noticed that Attr-DXL is interpreted
  1. when you open the module, even if you don't access an attr-dxl value
  2. when you modify the code and then access a value

That means if you attr-DXL is simply an #include statement, when debugging you have to modify the code and then modify and save the attr-DXL. Simply re-saving is inadequate so I tend to add or remove a space to the end of the #include statement. Thus att-DXL doesn't waste space based on how often it runs.

If you want to see some other interesting things, the open Modules and close them. We have some large modules which creates more than 1.000 memory leaks only be opening and closing them.